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ABSTRACT 

Attributes of teaching and learning contexts provide rich information about how students participate in learning activities. 
By tracking and analyzing snapshots of these attributes captured continuously throughout the duration of the learning 
activities, teachers can identify individual students who need special attention and apply different pedagogical actions to 
them. This paper describes the results of the work-in-progress study in exploring characteristics of fine-grained behaviors 
of learning mathematics in tablet-based e-learning activities. An experimental platform called SkyApp is built. Through 
SkyApp, teachers can create e-learning activities and track learning records of students after the delivery of the activities. 
SkyApp supports capturing, storing and analyzing of fine-grained behaviors of students. Pilot tests have been done in two 
primary schools for eight months. The review of the tests demonstrates the potential in performing learning analytics. By 
applying clustering algorithms on multiple learning metrics of marks, time and number of attempts for students in solving 
mathematics questions, classification of students by learning characteristics of performance and engagement can be 
formulated. 
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1. INTRODUCTION 

The infrastructure of e-learning in recent years allows teachers, school administrators, students and parents to 
monitor learning records that are generated during the delivery of learning activities, through which the 
insights of learners’ behaviors can be obtained by learning analytics (Becker, 2013). Learning analytics 
(Siemens, 2010) is defined as the use of intelligent data, learner-produced data, and analysis models to 
discover information and social connections, and to predict and advise on learning. Based on the results of 
learning analytic, teachers can identify students with learning difficulties so that strategic interventions can 
be made (Dawson, 2010). Researchers explore different technology and data sources to discover digital 
footprints of learners, such as sensors for capturing and analyzing eye-tracking, heartbeat, skin conductivity, 
gesture, voice and etc. Blikstein (2013) called this approach as multimodal learning analytics. Through 
automated, fine-grained and high-frequency data collection, this approach allows measurement of affective 
factors which include emotions of the learners. Rienties and Rivers (2014) review more than 100 recent 
studies of learning analytics in which the role of emotions is considered as one of the key drivers of 
e-learning. They identify approximately 100 different emotions that may have different levels of impact on 
learners’ attitude and cognition. In addition to learning emotion, researchers measure students’ behaviors of 
engagement (Richards, 2011) and motivation (Hershkovitz, & Nachmias, 2008) in e-learning activities. 

RELATED WORK: As classified by Wise (2014), learning analytics can be applied by the stakeholders 
at macro and micro level. At macro level, learning analytics supports the decision making process of 
administrators that are related to the institutional level and beyond. Siemens and Long (2011) refer this type 
of learning analytics as Academic Analytics. Based on the records stored in learning management systems 
(LMS) and virtual learning environment (VLE), visual graphical tools are built to show activity information 
of students, which include frequency of using resources, time spent per student in each resource, etc. Notable 
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examples of learning analytics tools that explore the records in LMS such as Matep (Zorrilla & Alvarez, 
2008). These platforms help monitoring learning records that are generated by the LMS that is adopted by the 
school for delivering learning activities. More recently, tools of learning analytics have been developed to 
perform analysis of log data in VLE (Agudo et al., 2013) and Khan Academy platform (Ruiperez et al., 
2014). These analytic tools are often used to determine summative learning data and identify “at risk” 
students, but they are pedagogy-neutral. As for supporting teaching and learning at the micro level, tools 
require addressing the challenges of capturing, analyzing and displaying learning data for improving 
pedagogical practices. For example, LOCO-Analyst (Jovanovic et al., 2008) obtains learning data from the 
online learning environment and presents the feedback data to teachers for evaluation. 

However, it is a heavy burden for the teachers of primary and secondary schools if they need to carry out 
the works of design, development, and delivery of e-learning activities by themselves. To tackle this 
challenge, teachers need a tool to capture, analyze and present the learning data. There are currently very few 
tablet-based mobile apps that keep track of on the fine-grained data about students’ inputs during e-learning 
activities. This paper introduces a new app, called SkyApp, as a platform to explore the possibility of using 
learning analytic to enhance the outcome of teaching and learning. Some parts of our work have been 
presented in CITERS 2016 (Shunt et al., 2016), and a detail technical report is posted on our university’s 
website (Hui et al., 2015). 


2. METRICS OF LEARNING BEHAVIORS 

SkyApp aims at helping mathematics teachers to deliver tablet-based e-learning activities. It allows teachers 
to import teaching and assessment contents in developing e-learning activity simply by file upload or 
photo-taking based on their existing teaching materials. The mathematics teachers of the participating schools 
create learning activities such as worksheets, quizzes, competitions, and homework for primary 4 to 5 (Ages 
9-10) classes with the help of SkyApp. The objectives of these e-learning activities align with the curricular 
goals defined in the schools’ teaching plan and annual subject plan. By measuring students’ input on tablet 
computers, it is possible to examine fine-grained students’ behaviors in detail. SkyApp offers facilities for 
students to provide answers or responses to the questions or assessment by typing, handwriting and adding 
emojis. SkyApp captures inputs such as time spent on each part of the e-learning activity and the details of 
the handwriting which are essential in supporting data analysis. 



Figure 1. Left - Entering Answer Box, Adding Emoji and Handwriting Inputs, Right - Worksheet 31 

The facility of Answer Boxes (Figure 1 -Left) helps to capture the inputs of learners. SkyApp captures 
the answers entered by students and performs automatic matching with the modeled answers. In addition, 
students can select and add emojis to the SkyApp, to express common emotions in participating the 
e-learning activity (Figure 1 - Left). In the form of worksheets presented in SkyApp, students can finish the 
questions under a specific topic, such as arithmetic operations of fraction, in each e-learning activity. 
Through the event logs of SkyApp, the inputs of users can be captured. There are two types of event logs, 
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namely independent and dependent events. An independent event occurs only at one specific instance in 
time, for example inserting an emoji by a student is recorded as an independent event. A dependent event 
whereas records information at multiple instances of time, starting by a start of event action and following a 
series of in-process actions until ending by the end of event action. For example, during inputting answers in 
the Answer Boxes of the questions Q1 and Q2 in a worksheet, inputs are recorded as dependent events. Idle 
Times and Answer Times of a user can be measured in answering the first two questions of the worksheet 
(Figure 2). 


Start 

worksheet 


startTyping endTyping startTyping endTyping 


Idle time of Q1 


Idle time of Q2 


Answer time of Q1 Answer time of Q2 

Figure 2. Example of Idle Times and Answer Times 

The learning metrics of Time, Attempts and Marks are used to perform analysis of learning data. Based on 
event logs, the measuring metric of Time is formulated as the sum of Idle Times and Answer Times of all the 
questions in a worksheet. By taking the maximum number of attempts in answering the same question in a 
worksheet, the measuring metric of Attempts is formulated. For the measuring metric of Marks , it is 
determined by the sum of marks assigned in each correctly answered question. 


3. DISCOVERING LEARNING BEHAVIORS BY CLUSTERING 

With the infrastructure of mobile learning in place, which includes WiFi network, tablet computers and 
educational mobile apps, primary and secondary schools can now adopt new models of e-learning that 
unleash the potential of applying new technology in education. Pilot tests are conducted with the participation 
of the mathematics teachers in the classes of two local primary schools. The tests were conducted mainly by 
a total of 64 students. All students are from one class of primary 4 and another class of primary 5 separately 
in the two primary schools. The test lasted 8 months in the academic year 2015-2016. The e-learning 
activities essentially include mathematics questions in the form of worksheets that are delivered as class 
exercises. 

To conduct the experiments, tablets of all the students and the teachers in the class are running SkyApp 
which are connected to the WiFi available in their schools. SkyApp is written in Objective-C that runs in iOS 
tablets. The data captured by SkyApp is structured in JSON format before sending to the back-end system for 
learning analytics. To perform analysis of the records captured, an analytical tool is built in Python to extract 
raw data from the database and convert them into CSV files. Data analytics can then be performed by calling 
statistical libraries through R programming. Analysis reports and classification results are then presented 
graphically in the dashboard by FITML. Clustering algorithms (Rokach, 2009) are used to find the data points 
represented in specific learning metrics that group together into a set of clusters. Patterns of learning 
characteristics can be identified by applying clustering algorithms on one or multiple learning metrics. The 
clustering method is called Complete-linkage clustering, which is one of the agglomerative hierarchical 
clustering. Literally, agglomerative method means that each object starts out as its own cluster, and close 
individuals or clusters are gradually merged until all individuals are grouped into a single cluster or any 
certain termination conditions are satisfied. Four criteria are used for identifying the best number of clusters, 
namely Silhouette, KL, CH and C-Index. A scientific way to study the relationship of clusters is to calculate 
the Euclidean distance between the center points of the clusters. Flowever, the most straightforward way is to 
generate a graph that can visually observe the distance between clusters. 

A typical experiment of one of the participating schools is described in this paper to demonstrate the 
effects of exploring fine-grained learning behaviors by using SkyApp in mathematics classes. Figure 3 shows 
the results of the students of the school in using SkyApp to complete Worksheet 31 (Figure 1 - Right) which 
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consists of 8 questions of fraction multiplication. Clustering algorithms are performed based on the metrics of 
Time, Attempts and Marks. The results in Figure 3 are presented in four quartiles of ranges of Marks, which 
are the most common ranges used by teachers and schools to represent students’ performance. The clusters 
identified in Figure 3 can be considered as meaningful patterns of learning characteristics of engagement and 
performance. As shown in Figure 3, Cluster D splits into two groups, one includes students 478, 484, & 503 
and the other includes Students 497 & 499. The students in this Cluster scored relatively low marks, tried 
relatively fewer attempts and took relatively short time in answering the questions. These students show 
possible signs of relatively low learning engagement and performance. Students of Cluster A and Cluster E 
show efforts in making extra attempts and spending more time in answering the questions, whereas the 
students of Cluster B and Cluster C show relative strength in learning performance among other students. To 
further distinguish the difference between Clusters B and C, different ranges of Marks can be applied to the 
presentation of Figure 3. The benefits of clustering of students’ learning characteristics to teachers are 
twofold: teachers can have an overview picture of the learning outcomes of the whole class in each e-learning 
activity, and teachers can have insights of the traits of learning of individual students. 

Marks: 0 “ 25% Marks: 25 ~ 50% 



Attempts Attempts 


Marks: 50 ~ 75% Marks: 75 ~ 100% 



Attempts Attempts 

Figure 3. Clustering by Attempts and Time in different ranges of Marks 


4. EVALUATION 

Based on the results of student surveys on 64 students at the end of test period, around 80% of students want 
to continue to use SkyApp in Math classes and around 69% of them agree that SkyApp makes the learning in 
Math classes more interesting to them. 10 students are interviewed to collect detailed comments on the 
usability of SkyApp. Besides the suggestions of new functions to SkyApp and improvements required to 
make SkyApp more effective, students expressed keen interests in the interviews that teachers can use 
SkyApp to offer incentives to them by giving out cartoon icons upon completion of e-learning activities. This 
shows that student engagement can potentially be enhanced by elements of gamification. 
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5. SUMMARY AND FUTURE WORK 

An experimental platform called SkyApp is developed to support tablet-based e-learning activities in learning 
mathematics with the capabilities in supporting learning analytics based on the fine-grained learners’ 
behaviors due to students’ inputs and responses. Initial efforts have been made in classifying learning 
characteristics of performance and engagement based on the results of the pilot tests. The next critical step is 
to investigate whether learning analytics can be applied to classify other learning characteristics such as 
motivation and emotion. Rigorous evaluations by quantitative and qualitative research methods are required. 

This study also brings up some questions that deserve further investigation: 1) Real-time learning 
analytics: the results show that it is technically feasible to perform clustering algorithms on learning metrics 
in real-time. The most time demanding part of the analysis is the computation time of running clustering 
algorithm and generating graphs. The time required is 6 to 10 seconds. It is feasible for teachers to view the 
results of the learning analytics right after each e-learning activity so that the results can help the teacher to 
revise the pedagogical actions of subsequent activities. 2) Personalized supports to students: through 
real-time analytics, teachers can revise their pedagogies and introduce specific design elements such as 
gamification elements to engage students, and provide scaffolding supports according to the pace and style of 
learning of individual students. We believe that with more participation in using SkyApp by teachers, more 
concrete personalized e-learning pedagogical strategies can be discovered. 
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